AN EXOTIC DEFORMATION 
OF THE HYPERBOLIC SPACE 

NICOLAS MONOD* AND PIERRE PY 



Abstract. We construct a continuous family of non-isomctric proper CAT(— 1) spaces 
on which the isometry group Isom(H n ) of the real hyperbolic n-space acts minimally 
and cocompactly. This provides the first examples of non-standard CAT(O) model 
spaces for simple Lie groups. We also classify all continuous non-elementary actions 
of Isom(H n ) on the infinite-dimensional real hyperbolic space. 
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1. Introduction 

Let Isom(H") = PO(l,n) be the isometry group of the real hyperbolic space H™ of 
dimension n > 2. The following theorem will be restated more precisely later in this 
introduction: 

Theorem A. There is a continuous family {Ct}o<t<i of proper CAT(— 1) spaces equipped 
with a continuous family of cocompact minimal isometric Isom(H") -actions, abutting to 
C\ = H". The spaces Ct are mutually non-isometric (even after reseating) and satisfy 
Isom(C t ) = Isom(H"). 

This stands in sharp contrast to the fact that geodesically complete CAT(O) model 
spaces for simple Lie groups are unique up to scaling, as proved in [5J Theorem 1.4] (and 
answers Question 7.2 loc. cit). 

The spaces Ct of Theorem [A] will appear as canonical minimal invariant convex sub- 
sets in the infinite- dimensional hyperbolic space H°° for a family of representations of 
Isom(H" ) on H°°. These representations will be deformations of the standard embedding 
of Isom(H ra ) into Isom(H°°) and likewise one can view Ct as a deformation of the stan- 
dard image C\ = H" C H°° arising as the limit of the composed standard embeddings 
H" C H ,l+1 C H"+ 2 . . . 

Accordingly, a large part of this article is devoted to the study of representations of 
Isom(H n ) into Isom(H°°). We shall indeed classify the (continuous, isometric) actions of 
Isom(H") on H°°. 

We now pass to a more detailed presentation of the content of this article. 

l.A. Context. Consider a real Hilbert space 3%? and a line L in Let B be the bilinear 
form on 34? determined by the quadratic form 

B(v,v) = \v L \ 2 -\v L ^\ 2 , 

where «l and v^± are the projections of v on L and L . We fix a non-zero vector e G L. 
As in the finite-dimensional case, define the space H°° by the hyperboloid model: 

H°° := {v G 34? : B(v, v) = 1, B(v, e) > 0}. 

The formula 

cosh(d H °°(a;,2/)) = B(x,y) 
defines a distance c?h~ on H°° which turns it into a complete CAT(— 1) metric space. We 
will denote by <9H°° the boundary of H°° , which can be identified with the Grassmannian 
of i?-isotropic lines in 34?, and by Isom(H°°) its isometry group. The study of this 
space, as well as that of certain infinite-dimensional symmetric spaces of finite rank, was 
suggested by Gromov in [20]. We refer the reader to [4] for a detailed study of the space 
H°°, and to |T5] [T5] for the case of higher rank symmetric spaces. This is the geometric 
viewpoint on the classical theory of Pontryagin spaces [24j [36l [33 [3H |39j [42] . 

An action g: G — >• Isom(H°°) of a group G on H°° is called elementary if it fixes a 
point in H°° U <9H°° or if it preserves a geodesic in H°° . If g is non-elementary, one can 
prove that there exists a unique closed totally geodesic subspace of H°° which is 
G- invariant and minimal for this property (see [1]). It is easily checked that an action 
g: G — > Isom(H°°) is non-elementary and satisfies = H°° if and only if the associated 
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linear action of G on Jtif is irreducible. In this case, we will simply say that the action of 
G on H°° is irreducible. 

It is a classical fact that any continuous isometric action of the group Isom(H n ) on a 
finite- dimensional symmetric space of noncompact type X preserves the image of a totally 
geodesic embedding H n «-» X . This is a particular case of a theorem of Karpclcvich and 
Mostow |26j [35] (see also [21], or [2] for a geometric proof in the hyperbolic case). The 
starting point of this article is the following fact, showing that the Karpelevich-Mostow 
theorem does not hold anymore in the infinite-dimensional context: 

There exist continuous irreducible isometric actions of Isom(H") on the 
space H°°. 

In other words, there exist irreducible representations of Isom(H") on a Hilbert space 
which are not unitary but which instead preserve a quadratic form of index 1. More 
generally, there also exist irreducible representations of the group Isom(H") on a Hilbert 
space which preserve a quadratic form of index p, for certain values of p > 1 (see Sec- 
tion [5]B]). These facts are well-known to representation theorists and can be found, for 
instance, in [25] . 

These representations were put in a geometric context in |14j where the authors showed 
how actions of PSL2(R) on H°° arise in the study of fundamental groups of compact 
Kahlcr manifolds. There are other examples of groups acting on an infinite-dimensional 
hyperbolic space: for the Cremona group, see [7], for automorphism groups of trees, 
see [201 Chap. 6] and g]. 

l.B. Results. If g: G — > Isom(H°°) is an action of a group G on H°°, we will denote by 
£ g the associated translation length function, i.e. the function defined by 

^(flO = inf dn°°{Q(g)(x),x). 

We will denote by £h™ the translation length function for the standard action of Isom(H n ) 
on H". We can now state our first main result, which classifies the (continuous) irreducible 
representations Isom(H") — s- Isom(H°°). 

Theorem B. 

(1) Let g: Isom(H n ) — > Isom(H°°) be a continuous non- elementary action. There 
exists t G (0, 1] such that i g (g) = ^h™(<?) for all g € Isom(H"). Moreover, t = 1 
if and only if g preserves an n-dimensional totally geodesic subspace o/H°°. 

(2) For each t £ (0, 1) there is, up to conjugacy in Isom(H°°) ; exactly one irreducible 
continuous representation gt'. Isom(H") — > Isom(H°°) such that £ et = t£nn. 

Thus, all irreducible continuous representations g: Isom(H n ) — s- Isom(H°°) arise from 
an explicit one-parameter family of representations. It could be interesting to search for 
another proof of this using the machinery of Lie algebra representations [49j . 

In what follows, we will denote by gh yp the Ricmannian metric with constant curvature 
— 1 both on H" and on H°°. We also denote by du." and du°° the associated distance 
functions on H™ and H°° respectively and by H°° (rcsp. H") the geometric bordification 
H°° U 9H°° (resp. H" U <9H n ), endowed with the cone topology (3] II.8]. 

Theorem C. There exists a smooth harmonic g t -equivariant map ft'. H n — > H°° with 
the following properties. 
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(2) The image of ft is a minimal submanifold of curvature 



f(t+n-l) 



and we have 




t(t + n-l) 



it 



9hyp- 



(3) The map ft extends to a continuous map from H" to H°°. 

In fact, it will be clear from the model for g t that f t is the only equivariant map H" — ^ H°° 
(Lemma !3.8l below). When n = 2, the existence of the family g t and some of its properties 
were established in (T4"] . 

Before describing our next result, let us recall a theorem from [8|. Let k be a local 
field and G be an absolutely almost simple simply connected fc-group. Let X mo d e \ be 
the Riemannian symmetric space or Bruhat-Tits building associated with G(fc). The 
following is proved in [5] (see Theorem 7.4 in [3] for a more precise statement): 

Let X be a noncompact CAT(O) space on which G(k) acts continuously and 
cocompactly by isometries. If X is geodesically complete, then X is isometric 
to X m odei (possibly rescaled). 

Without the assumption of geodesic completeness, it is known that this result can fail 
when k is non-Archimedian. More precisely, X need not contain a closed, G(fc)-invariant 
convex set isometric to X mo dci (see Example 7.6 in [8]). When k is Archimedian, it was 
not known whether the theorem above remains true without the hypothesis of geodesic 
completeness (see Question 7.2 in [3]). The next theorem shows that this is not the case, 
at least for the group Isom(H"). Indeed, we shall construct exotic cocompact spaces X, 
which are even nice enough to be CAT(— 1) and without branching geodesies, since they 
arise as convex subspaces of H°° . 

It is known that any non-elementary isometric group action on H°° admits a unique 
minimal non-empty closed convex invariant subspace C C H°° (see Section I4.AI below). 
In the case of the action g t (t € (0, 1)), we denote it by Ct and it turns out to provide 
our "exotic" space. In what follows, we will use the following convention: C\ will be the 
hyperbolic space H" and f\ the identity map H" — > C\. 

Theorem D. For any t £ (0, 1], the CAT(— 1) space Ct is locally compact and the action 
o/Isom(H") on Ct is cocompact. 

Moreover, the spaces Ct and Ct' are isometric only if t = t' ( even upon reseating the 
metric). 

The following proposition describes Ct more precisely. 

Proposition E. Let t e (0, 1]. 

(1) The set Ct coincides with the closed convex hull of ft(H n ) ■ 

(2) The set Ct coincides with the closed convex hull of ft{dH n ) and dCt = ft(dH n ). 



(3) Isom(C t ) = Isom(H"). 
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Next, we study the continuity with respect to t of the metric space Ct- Recall that 
there is a natural topology called the pointed Gromov-Hausdorff topology on the class of 
all pointed metric spaces. We refer the reader to [19l chap. 3] or to Section 14. Bl for its 
definition. If G is a fixed topological group, a metric space endowed with a continuous 
isometric action of the group G will be called, for short, a metric G-space. On the class 
of all pointed metric G-spaces, there is a natural refinement of the pointed Gromov- 
Hausdorff topology which takes into account the G-action (see for instance Q~5]). We 
will define precisely a suitable version of this refined topology in Section 14. Bl and call 
it the strong topology on pointed metric G-spaces (rather than the "equivariant pointed 
Gromov-Hausdorff topology" . . . ) . 

The spaces Ct will always be pointed at /t(o), where o is a given point of H™. 

Theorem F. The map t ^ Ct is continuous on (0, 1] for the strong topology on the 
class of pointed Isom(H n ) -spaces. In particular, as t — > 1, Ct converges to H' 1 and the 
diameter of Gt/Isom(H") goes to zero. 

Concerning the behavior of the spaces Ct as t goes to 0, we refer the reader to Sec- 
tion 14. CI for a discussion. 

The article is organized as follows. In Section [2j we prove completely Theorem [5] 
except for the existence of the family gt . In Section [3l we recall some classical facts 
concerning the spherical principal series of representations of Isom(H n ) and study the 
actions of Isom(H n ) on H°° which arise from the spherical principal series. We then 
complete the proof of Theorem [B] and prove Theorem [C] In Section [4] we study the 
spaces C t '. we prove Theorem iDl Proposition [E] as well as Theorem IF1 Finally, Section [5] 
contains further results: Section [5 . Al deals with actions of automorphism groups of trees 
on H°°, already studied in [¥| and Section T5.BI starts the study of actions of Isom(H n ) 
on certain higher rank infinite-dimensional symmetric spaces which also arise from the 
spherical principal series. 

Acknowledgements. We would like to thank Serge Cantat, Thomas Delzant and Robert 
Stanton for several useful conversations during the preparation of this work. 

2. Classification of non-elementary actions 

In this section, we start the classification of non-elementary (continuous) actions of the 
group Isom(H n ) on H°°. The strategy of the proof is similar to the one in [4] where an 
analogous result is proved for actions of the automorphism group of a regular tree on H°° : 
we first prove that the action is determined by its restriction to a parabolic subgroup and 
then study the action of parabolic subgroups. The second step of the proof differs from 
the case of trees since parabolic subgroups have distinct structures in Isom(H") and in 
the automorphism group of a tree. 

We will use the fact that, as in the finite-dimensional case, isometries of H°° fall into 
three types: elliptic, parabolic or hyperbolic (see [I] for more details). 

2. A. Restriction to a parabolic subgroup. We start this section by introducing some 
notations. We consider the vector space R™ +1 endowed with the bilinear form B defined 
by: 

o/ ^ 2 _ 2 _ 2 

_D \ X •) X J — X ^ ' ' ' " 
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As usual, the hyperbolic space H" is described as: 

H" := {x G R™ +1 , B(x, x) = > 0}; 

the group Isom(H n ) is the group of linear maps g: R' l+1 — > R ,1+1 which preserve B 
and satisfy g(H n ) = H". We denote by ei, . . . e n+ i the canonical basis of R™ +1 and by 
£1 = ^(lj 1,0,.. .) and £2 = ^7f (1, — 1; 0, • • •)• The vectors £i and £2 are i3-isotropic. In 
what follow we will rather use the basis (£1, £2, £3, ■ ■ ■)• In particular all the matrices we 
write below are decomposed into blocks corresponding to the decomposition 

R' l+1 = Ra © R6 © U, 

where U is the vector space generated by the family (ej)3<i<n+i, that we will identify 
with R n_1 . Hence they have the following form: 




where a, /3, 7, 5 are real numbers, V\ and v-i are vectors in R n_1 , l\ and I2 are linear forms 
on R' l_1 and M is an endomorphism of R n_1 . 

If A G R^, A G 0(n — 1) and v € R™ -1 , let g\ tV) A be the element of Isom(H") whose 
matrix has the following form: 




Here (■, •) stands for the standard inner product on R™ -1 . The group P = {g\, Vt A} is 
the stabilizer of the line R£i in Isom(H"). It is isomorphic to the group Sim(R™ -1 ) of 
similarities ofR n_1 : the map which takes Qx^v.A 

to the similarity 

x 1 — > XAx + Xv 

is an isomorphism from P to Sim(R" _1 ). 

Note that the isometry g\, v ,A is parabolic if and only if A = 1 and v (£ Im(A — Id). 
Hence a parabolic element gi, v ,A is always conjugated inside P to an element of the form 

( 1 |K| 2 («',•) \ 
01 00 

v' Id 
\ B J 

with B — Id invertible and v' 7^ 0. In particular, one sees that if g\ jV ,A is parabolic, there 
is always another parabolic element h G P which commutes with gi, v ,A and which is 
conjugated in P to its square. Finally, we will denote by a the involution represented by 
the matrix 

(2.i) 

where J\ is the diagonal matrix with coefficients (—1,1,...). The group Isom(H") is 
generated by a and P. 
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Prom now on, wc fix a continuous non-elementary representation 

q: Isom(H") — > Isom(H°°). 



Let xq be a point in H°°. Since g(Isom(H™)) does not fix any point of H°° and thanks 
to a classical argument going back to Cartan (see [3], Chapter II. 2), the function 



has to be unbounded. It is then a well-known result that this implies that the function 
<f e ,x is proper. The reader will find a proof of this fact in (in a more general context). 
The properness of the displacement function for actions of simple Lie groups of rank 1 on 
Hilbert spaces was proved by Shalom in |45| . From the properness of tp e ,x a , one deduces 
easily the following: 

Proposition 2.1. The action g preserves the type, i.e. the image g(g) of an elliptic 
element of Isom(H") (resp. parabolic, hyperbolic) is an elliptic isometry of H°° (resp. 
parabolic, hyperbolic). 

Proof. Since elliptic elements of Isom(H ra ) are contained in a compact group, their image 
under g has to be elliptic. Since the function ip g _ Xo is proper, the image by g of a non- 
elliptic element in Isom(H") has to be non-elliptic. As noted above, if g E Isom(H") is 
parabolic, there exists another parabolic element h € Isom(H") which commutes with g 
and which is conjugated to its square. The isometrics g(h) and g(g) have the same type, 
hyperbolic or parabolic, but g(h) cannot be hyperbolic, being conjugated to its square. 
Hence g{h) and g{g) are parabolic. 

From this, one already sees that the group g(P) has a unique fixed point [771] in the 
boundary of H°°. Indeed the elements g{gi, v ,\d) for v <G R" _1 are commuting parabolic 
isomctries, hence have a common fixed point. Since the group formed by the gi^id is 
normal in P. [771] must be fixed by all of g{P). 

Next, let g £ Isom(H n ) be hyperbolic. Up to conjugacy and up to replacing g by a 
power, we can assume that g lies on a I-parameter subgroup of the form g t = g^ ^,A t - 
If g were parabolic, the entire 1-parametcr subgroup gt would have a unique fixed point 
£ in <9H°°. Note that the point £ would coincide with the fixed point at infinity [rji] for 
the whole group g(P). In particular the I-parameter group ge'.ojd would admit £ as a 
unique fixed point. Since the involution a normalizes the latter group; £ would be fixed 
by a. Finally, being fixed by a and P, £ would be fixed by all of Isom(H"). This is a 
contradiction since g is nonelementary. □ 

We will still denote by [771] the unique fixed point of g(P) in <9H°°. We now continue 
our study of the action of P. Since g(P) does not preserve the horospheres centered at 771 
(it contains hyperbolic isometries), it follows from Proposition 4.3 in [3] that there exists 
a unique closed totally geodesic subspace Hp which is P-invariant and minimal with this 
property. We can actually describe this space. Let 772 be an isotropic vector such that the 
lines [77!] and [772] are the two fixed points of the hyperbolic isometrics p(gA,o,id)- Assume 
that 5(771, 772) = I. Let E = 77^ n 77^. We will represent the isometries g(g) for g £ P by 
matrices associated to the decomposition Jtff = R771 © R772 © E. We also denote by (-, •) 
the scalar product induced by —B on E. One can then write: 
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where x '■ R-+ R-+ is a continuous homomorphism, 7r is an orthogonal representation of 
P in E and the map 

c: P — > E 

satisfies that u(X, v, A) := x(A)c(A, v, A) is a cocycle for the linear representation \ <& 7r. 

Let 

F:=5pan{c( ff ), ff eP} 
be the closed vector space spanned by the vectors c(g) for g E P. Then it is easy to see 
that R771 © R772 © U is p(P)-invariant and minimal with this property. The space Hp is 
thus the intersection of R771 © R772 © V with H°° . The map c enjoys more properties. The 
fact that <7A,o,id and g^fi^A commute implies that Q{g ll ,Q, a) preserve the axis of £>(ffA,o,id)) 
we thus have: 

c{n,0,A) = 0. 

A simple computation shows that this implies that 

(2.ii) X (A)c(A > t) > A) = c(l > A« > Id). 

Hence: 

Lemma 2.2. The space V is the closure of the vector space generated by the c(l,u,Id) 
for v en™- 1 . □ 

Proposition 2.3. One has x(A) = A* for some < t < 1. Moreover, if t = 1, then P 
preserves an n- dimensional totally geodesic subspace 0/H 00 . 

Proof. Let us write x(A) = A* for some non-zero real number t. Let g s = g e s ,o,id- K 9\,v,A 
is in P, the subset 

\9s9\,v,A9-s = 9X,e s v,A}s<0 

of P is relatively compact. Hence, for fixed A, v and A, the vectors {c(A, e 3 v, A)} s <q must 
be contained in a compact subset of E. Since |c(A, e s w, A)| = e st |c(A, w, A)\, and since 
c(A,w, A) is non-zero when g\ >v ,A is parabolic, we must have t > 0. 

Now let /(u) := c(l,u, Id). Using the cocycle relation for u = xc, one proves that: 

(2-iii) f(Xv) = x(A)7r( ffW d)(/(w)), 

(2-iv) /(« + «;) = /(«) + 7r( ffl ^, Id )(/H). 

The first equation implies that |/(nw)| = \(n)\f(v)\, on the other hand the triangle 
inequality combined with the second equation implies that |/(m>)| < \n\ ■ |/(u)|. This 
implies t < 1. If t = 1, we must have 7r(5i,«,id)(/(w)) = f(v). We shall write from now 
on S v '■= """(gi.uJd) to simplify notations. Note that S v + W = S v a S w . According to the 
previous remark we have: 

S Vl +v 2 (f(vi + v 2 )) = f(vt + v 2 ). 
Using the cocycle relation (|2.ivj) on / we deduce: 

S V iS V2 (f{v 1 ) + S vi f{v 2 )) = f(v 2 ) + ^(/(ui)), 

and then 

S 2vi (.f(v2)) = f(v 2 ). 

Hence S w (f(v)) = f(v) for all v,w, € R™ -1 . Using Equation (|2.ivl) again, this implies 
that /: R" _1 — > P is linear. It has to be injective; indeed if f(y) = and u / 0, the 
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isomctry g(gi. v ,id) is elliptic, in contradiction with the fact that g preserves the type. 
Hence V has dimension n— 1, and g preserves the linear space R771 © R772 © V. □ 

Note that if V contains a non-trivial subspacc on which R' 1 " 1 ~ {<7i „ id} acts trivially 
we must have t = 1. Indeed let p be the orthogonal projection from 1/ to the space of 
fixed points of R n . Then po f is a homomorphism, which is non-trivial since the image 
of / generates a dense subspace of V. This implies that t = 1. Hence, if i < 1, the 
orthogonal representation it does not have R Il_ ^invariant vectors inside V . 

Since we have proved that the action g preserves the type, the previous proposition 
implies that the translation length function associated to g satisfies l g — t£n n , thus 
proving the first part of Theorem [B] From now on, we will assume that the action g is 
not only non-elementary but irreducible. We will prove that g is completely determined 
by the number t appearing in the previous proposition. 

Proposition 2.4. The subspace V is equal to E and the representation g is determined 
by its restriction to P. 

Proof. We have already noticed that P together with the involution a generate the group 
Isom(H n ). Here, as in [3], we use the relations between a and P to prove that g{a) is 
determined by the restriction of g to P. More precisely, we consider the product 

.9 : = a ' ffA.u.Id ■ cr • g^w, id ■ g\,v,id ■ °~, 

where A, p € R* and v G R rl_1 — {0} arc fixed and w will be determined. As in the 
definition of the involution o~, we will denote by J\ the orthogonal reflexion of R™ -1 
fixing the hypcrplane orthogonal to the first vector of the canonical basis. A long but 
simple calculation shows that if w is chosen so that 

w -2JiO) 
J + V ~ A/iM 2 ' 

then the element g above belongs to P and one actually has g = g v . Ut A for 

2 

V = To 1 19 1 u = XnJx(v), A = s u oJ 1 . 
Here s u is the reflection orthogonal to the vector u. We now use the relation 

g{g) = K CT MsA,^id)e(CTM5/i,iMdMsA,MdM a ') 

to determine q((t). Since g(o~) interchanges the two fixed points at infinity of p(gA,o,id)) 
we can write: 

g(a) = 

We first compute the product g(o~)g(g\ -v jd)g(o~)g(gp,, w .id)g{g\.v,id) ano - ^ n( i that it has 
the following form: 

X (/i) ^\c(X^f+v,ld)\ 2 * 

i^ X (A 2 /i)|c(A, V ,Id)| 2 

yx(A/i)7r(cr)(c(A, v, Id)) * * 






v- 1 





V 
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Hence g(g), which is obtained from the matrix above by multiplying on the right by g(a) 
reads: 

%vx(l*MXli,$ +v,ld)\ 2 

* x(VMo")(c(A,v,Id)) * 

But this must be equal to the matrix 

x(v) 5X(»?)|c(77,u, A)\ 2 x(il)(c(il,v>,A),Tr(Ti,u, A)(-)) 

xiv' 1 ) o 

c(j],u,A) w(r),u,A) 

This shows that i^x(A t )l c (A/i, t + ^Id)| 2 = xiv) = x(2/A 2 ^|w| 2 ), hence v is determined 
by the restriction of g to P. It further shows 

7r(cr)(c(A, v, Id)) = x(AAi) _1 c(r7, u, A). 

Since the vectors c(A, v, Id) generate a dense subspace of V, this implies that n(a) pre- 
serves the space V, hence £>(ct) preserves the space R771 © Rf?2 © V. Since g was assumed 
irreducible, this implies that 

= R771 © Rr? 2 © V, 

in particular V = E. Moreover 7r(<r) is determined on a dense subspace of hence 
everywhere on E. This completes the proof of the proposition. □ 

We must now determine completely the restriction of the representation g to P, from 
the number t only. Observe that the stabilizer of [rjx] in Isom(H°°) is isomorphic to the 
group Sim(-E) of similarities of the Hilbcrt space E. Hence, we must study homomor- 
phisms 

P — > Sim(£), 

subject to the various constraints we just collected. We will do this in the next paragraph. 

2.B. Affine representations of the group Sim(R'). In this section, we work directly 
with the group Sim(R') of similarities of R'; applying the result of this paragraph with 
I = n — 1 allows to conclude the proof of Section T2.A1 We denote by S\, V ,A the map 

R' > R 

x 1 — > XA(x) + v 

where, once again, A S Rl, A G 0(7) and v € R'. We start with a representation 
at/. Sim(R') — > Sim(^f), where Sim(^f) is the group of similarities of a real Hilbcrt 
space Jif. We write 

at(S\. v ,A)(x) = A t 7r(S , A ,„ i A)(^) + c(Sx,v,a) (x G Jf) 

with 7r orthogonal. Thus n is an orthogonal representation and c is a 1-cocycle for the 
representation it © \ t , where Xt is the character of Sim(R') defined by Xt{S\, v ,A) = X* . 
We shall make the following standing assumptions in view of the results of the previous 
section: 

(1) 0< t < 1, 

(2) at preserves the type, 

(3) 7r has no R -invariant vectors, 

(4) the vector space generated by the vectors (c(5i ) „ ! id))^gRi is dense in ffl 1 , 

(5) c(5a, u ,a) = c{Si, v ,id) (this condition can always be achieved up to conjugacy). 
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Note that the last equation is equivalent to identity (|2.ii[) (remembering that the natural 
isomorphism bctwwcn the group P considered in the previous section and the group 
Sim(R™ _1 ) takes the element gx,v,A to the map x i-> XAx + Xv). 

The purpose of this section is to prove that a representation a t satisfying the conditions 
above is completely determined by t. 

We will denote by ttq be the unitary representation of Sim(R') on L 2 (R ( ) defined by 

n (S x , v , A )(f)(y) = \h i ^f(XA- 1 y). 

We will need the following information about ttq: 

Lemma 2.5. The space 

H^Sii^R^x^Tro) 

is 1- dimensional, generated by the cohomology class of the cocycle defined by 

_ e i(v,v) _ i 

c(Sx, v ,a)(v) = , „, i ■ 

\y\t+2 

Notice that c ranges indeed in square-summablc function classes: the condition t > 
ensures the integrability in a neighbourhood of infinity and the condition t < 1 ensures 
the integrability in a neighbourhood of the origin. 

Proof. Any cohomology class in H 1 (Sim(R i ), \t ® tto) can be represented by a cocycle 
which vanishes on the subgroup {S^o.a} — Rf x 0(7) of Sim(R z ). Let /3 be such a 
cocycle. The fact that /3(S\fi a) = implies that /3 satisfies Equation ([5]) above. We 
choose a non-zero vector b E R z . Let g be the measurable function on R' defined by: 

9\V) - e i(y, b ) _ l ■ 

Note that, because of the cocycle relation, g does not depend on the choice of b. Then 
the equation 

P(S\fi,\Ad+b,AB) = /3(S\^,a) + X t Tr c (S\ t b, A )((3(S tl ,d,B)) 

(which says that f3 is a cocycle) becomes 

^(y,\Ad+b) _ -Q g ( y j = ( e i(y,b) _ ]_) g ( y ) 

+X t+l e i(y,b) ^gi(\A~ 1 y,d) _ A g^A^y), 

which yields, after simplification: 

g(y) = X^giXA-'y). 

By taking A = 1 and letting A vary in 0(1), we see that g is constant on the unit sphere 
of R'. The equation g(y) = X t+ ^g(Xy) then implies that 

a 



g{y) 



for some constant a. In other words, j3 is proportional to the cocycle c. This proves that 
the group H 1 (Sim(R i ), Xt®^o) is generated by the cohomology class of c. We still need to 
check that this class is non-zero. Observe that c is the formal coboundary of the function 
g. Using the calculations above, one checks that any function / <G L 2 (R Z ) satisfying 
P(S\,v,a) = X t TTo(S\ iVj A)(f) ~~ f has to be proportional to g. Since no non-zero multiple 
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of g is square intcgrable on R/ this implies that / = 0, hence j3 = 0; a contradiction. This 
proves that the cohomology class of c is non-zero. □ 

We now study the complexification of the orthogonal representation ir. 

Proposition 2.6. The complexification ttc of tt is isomorphic to ttq; therefore the space 
H 1 (Sim(R'), xt <8 7Tc) is 1- dimensional. 

Proof. According to a theorem of Mackey (Theorem 14.1 in [29]), the representation ttc 
can be constructed in the following way. Fix a unit vector e in R ( and identify its stabilizer 
with 0(7 — 1). Then, there exists a unitary representation r\ of 0(1 — 1) on a Hilbert space 
U such that ire is described as follows. Choose a measurable section s : S i_1 — > 0(1) such 
that s(v)e = v for v £ S 1 . We define an element j3(A,v) £ 0(1 — 1) by 

s(A- 1 v) = A- l s(v)fi{A,v). 

Note that (3 satisfies the cocycle relation 

P{AB, v) = p{A, v)0(B, A~ 1 v). 

Then ttc is, up to isomorphism, the representation on L 2 (R Z , U) defined by 

(*c{S x , v ,a) ■ f) (v) = x^ v >nip{A, fy) (f(XA- 1 y )) . 

\y\ 

Observe that, as a representation of R ; , ire is isomorphic to pir® (where p is the dimension 
of the space U). 

Lemma 2.7. As representations ofW, He and ir are isomorphic. 

Proof of Lemma \2.7\ We assume by contradiction that p > 2. Up to passing to a sub- 
space, we can assume that p = 2, so that L 2 (R ( , U) ~ L 2 (R ( )©L 2 (R'). We can decompose 
the cocycle c arising from the affine representation a t according to the previous decom- 
position and write c = (c%, C2). The argument from Lemma |2 . 5 1 implies that C\ and C2 are 
both proportional to c: there exist constants and 02 such that Cj = afc. This implies 
that any vector 

v = (v!,v 2 ) £ L 2 (R l 7 U) ~L 2 (R ; )®L 2 (R Z ) 
which is in the vector space generated by the family (c(Si jVi id)) v ^Ri satisfies 02^1 — a\vi = 
0. This contradicts our standing assumption (jlj. □ 

According to the previous lemma, we must have p = 1 and U ~ C. This leaves us two 
possibilities for the representation 77: it is either the trivial representation or the character 
given by the determinant (this possibility occurs only if I > 2). In the second case, we 
observe that if we choose the section s above to take its values in SO(Z), the cocycle (3 
satisfies rj(f3(A,v)) = dct(A). 

We will see that only the case where 77 is trivial occurs. To do this, we repeat the 
calculations from Lemma 12.51 taking into account the presence of the character r\. We 
choose again a non-zero vector b £ R' and let g be the measurable function on R ; defined 
by: 

, s _ c(Si. b j d )(y) 

3\V) e i( y ,b) _ 1 ■ 

Assume first that 77 is the determinant. Then the cocycle equation for c leads to the 
equality 

g(y)^det(A)X t+ ig(XA- 1 y). 
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By taking A = 1 and y in the unit sphere one gets: g(Ay) = det(A)g(y) for A in 0(1). 
Since SO(l) already acts transitively on the sphere (since I > 2) g should be constant 
(a.e.). But then the equation g{Ay) = —g{y) for A £ 0(1) — SO(Z) implies that g is 
zero, a contradiction. Hence rj is the trivial character. This implies that ttc — ""o as 
representations of Sim(R'). According to Lemma [231 the space H 1 (Sim(R / ), \t ® ttc) is 
1-dimensional. □ 

Having described the complexification ttc of tt, we now go back to tt itself. 

Lemma 2.8. Let I: L 2 (R') — > L 2 (R') be an antilinear involution which commutes with 
the action o/Sim(R ). Then I — e 1 " 9 Iq for some i) £ R, where Iq is the involution defined 
by: 

(iof)(y) = f(-y). 

Proof. The map Iq defined above is obviously an involution satisfying the conditions of 
the lemma. We want to prove that this is the only one, up to multiplication by a complex 
number of modulus 1 . If / is another such involution, we can write I = J o Iq where J 
is complex-linear and commutes with the action of Sim(R ). But any bounded complex 
linear operator which commutes with the action of R ( < Sim(R') is the multiplication by 
a function h £ L°°(R Z ) (this can be seen as the commutative case of the commutation 
theorem of Murray and von Neumann). Hence: 

(If)(y) = h(y)J(-y). 

The fact that I commutes with the action of Sim(R') implies that h is constant almost 
everywhere and the condition I 2 = Id implies that \h\ 2 = 1 almost everywhere. This 
completes the proof of the lemma. □ 

Now, for each real number i9, the fixed point set of e 1 " 6 Iq is isomorphic (as an orthogonal 
representation of Sim(R )) to the fixed point set of Iq, via the map / \-> e'~ f. This 
implies that any two orthogonal representations of Sim(R') having ttq as complexification 
are isomorphic. The cocycle c defined above take its values in the fixed point set of Iq and 
the fact that H 1 (Sim(R ), Xt (8 ttq) has dimension 1 implies that H 1 (Sim(R'), \t ® tt) has 
dimension 1 also. This proves that, up to isomorphism, there is a unique representation 
at : Sim(R') — > Sim(^) satisfying the condition of the beginning of the paragraph. 

This concludes the proof of Theorem up to the existence of the representations §t 
that will be discussed in Section [3] 

3. The principal series and actions on hyperbolic spaces 

To construct the family of representations g t , we first need to recall a few well-known 
facts from representation theory (which can be found, for instance, in [231 \T7\ H31 144] ) . 

3. A. Preliminaries from representation theory. We continue to use the notations 
from paragraph 12. Al Let o = (1,0,..., 0) £ H" and denote by K its stabilizer in 
Isom(H"). We denote by Vol the unique normalized i<T°-invariant volume form on <9H". 
Let Jac(<7)(-) be the Jacobian of g £ Isom(H") for the volume form Vol; i.e. (<7*Vol)& = 
Jac(g)(6)Voh for b £ dH n . If one thinks of Vol as a measure, |Jac(g)| = dg~ Vol/dVbl. 
Since we have |Jac(g)| = \Ja,c(kg)\ for k £ K, the function | Jac(5 _1 )(6)| descends to 
H™ x <9H n ; this is the Poisson kernel. One has (see [25]): 
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(3-i) ^-r m= ^L.y\ 

There is a slight abuse of notation in the formula above: on the left-hand side b represents 
a point of the boundary of H™ i.e. a B-isotropic line in R n+1 . To compute the right-hand 
side, we can choose an isotropic vector representing the line b. 

The (spherical) principal series is the family of representations tt s of Isom(H n ) on the 
space L 2 (<9H n ), parametrized by a complex number s and defined by: 

7T s (g)-f=\^(g- 1 p +s fog- 1 . 

Remark 3.1. Here L 2 (5H") stands for the space of complex-valued, measurable, square- 
integrable function classes on <9H" with respect to Vol, although we will soon restrict to 
real-valued functions, in which case we write L 2 (i9H n ,R). 

If /11/2 G L 2 (<9H"), we will write (/i,/2) = J 9H „ A/2- A simple calculation shows 
that (7r s (/i), 7r_s(/2)) = (/lj/z); hi other words the representations ir s and 7r_s are duaZ. 
It is known (see for instance 48 ) that the representation tt s is irreducible whenever 
s ^ ±(| + ^) for all k G N. 

Before going further, let us introduce a second model for the representations ir s . We 
will need to use a KAN decomposition for the group Isom(H"). We still consider the 
two isotropic vectors £1 and £2 considered in paragraph 12. Al 

£1 = -±=(1,1,0,...), 

6 = ^(1,-1,0,...), 

and we still denote by P the stabilizer of the line R£i in Isom(H"). Elements of P are 
denoted by g\, v ,A where A G Rl, v £ R™ _1 and A E 0(n — 1). Every element g of 
Isom(H n ) can be written uniquely as a product: 

9 = k ■ g\, v ,idi 

(where of course k, X and v are functions of g but we will not indicate it in the notation). 
Let M be the centralizer of the 1-parameter group {(?A,o,id}A in K. We can now define 
the second model for 7r s . 

Let V s be the space of measurable function classes 

F: Isom(H" ) — > C 

such that: 

• F(gmgx,v,u) = F(g)X~^ +s}(n ^ 1) , for g e Isom(H n ), m G M, A G R^ and 
v e R" Jl , 

• the restriction of F to K is square-integrable. 

Denote by dk the normalized Haar measure on K. The space V s , endowed with the norm 
I • I defined by 

|F| 2 = / \F{k)\ 2 dk, 

J K 

is a Hilbert space. The group Isom(H n ) acts on it by translation; the element g G 
Isom(H") acts by: 

(g-F)(h)=F(g- 1 h). 
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The following proposition relates the previous representations. 

Proposition 3.2. The representation tt s is isomorphic to the representation o/Isom(H n ) 
on V s . 

Proof. Note that the map k H> [fc(£i)] induces an identification between K/M and <9H". 
If F <E V s , its restriction to K descends to a square-integrable function on K/M ~ dH n . 
This defines a map from V s to L 2 (9H") which is easily seen to be a surjective isometry. 
We will denote by 

/ e L 2 (dH") ^F f EV s 

the inverse of this isometry. The point of the proof is to check that the action of Isom(H") 
on V s by translations identifies with the representation ?r s under this isomorphism. This 
can be seen as follows. Let / € L 2 (9H"), we will temporarily write g ■ f for the function 
which corresponds to Ff(g~ 1 -) under the isomorphism above. We want to check that 
g ■ f = i" s (<?)(/). Let b = [fc£i] be a point in the boundary of H". Let 

g~ k = k'g\', V ',id 

be the Iwasawa decomposition of <? _1 fc. A simple computation using Equation (|3.ip shows 
that |Jac(ff- 1 )(6)|*+ s = (A') _ ^+ s )(™- 1 ). One then has: 

(9 •/)(&) = F f (g^k) 

= F f (k')(\')-^+ s X n -V 
= |Jac(. 9 ' 1 )(6)|Hv( 5 - 1 6), 

where, in the last step, we used the fact that g~ 1 b = ,g _1 fc^i is equal to fc'^i. This proves 
that g ■ f = n s (g)(f), as desired. □ 

From now on, we will assume that s is a real number. We seek to construct a sesquilin- 
ear form on the space L 2 (3H"), invariant by the representation ir s . Since ir s and 7r_ s 
are dual, it is enough to construct an operator L s of L 2 (9H") which intertwines the 
representations n s and 7r_ s , i.e. which satisfies: 

L s o ir s (g) = 7r_ s (g) o L a . 

It will be more convenient to use the model V s to describe the operator L s . We first give 
a "formal" definition as follows. If / G V s and / is continuous, we consider the function 
L s f defined by: 



L s f(x) = / f(xgi. v jd<y)dv, 

where dv stands for Lebesgue integration on R™ _1 and a is the involution defined in (|2.ip . 

The following proposition is classical; the reader will find its proof in (for instance) [47], 
Proposition I page 91. 

Proposition 3.3. Assume that s > 0. Then if f € V s is continuous, the integral defining 
L s f is convergent and defines a continuous function in V- s . Moreover, the operator 
L s : V s — > V-s is continuous. 

From now on, and in view of this result, we will always assume that s > 0. Note that, 
by its very definition, the operator L s intertwines the actions of Isom(H") on V s and 
V- s : an element g € Isom(H n ) acts by left translation and the integral defining L s is a 
convolution on the right. According to Proposition l3.2[ we can think of L s as an operator 
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from L 2 (<9H") to itself (which intertwines ir s and 7r_ s ). We can thus define an invariant 
bilinear form (•, -) s on L 2 (9H n ) by: 

(f,g) s = [ fLTg. 

To continue further, we need to know whether the quadratic form (•, -) s is of finite index 
and how its signature depends on s (here by the index of a quadratic form we mean the 
maximal dimension of an isotropic subspace). This result is known and due to Johnson 
and Wallach [25] (for n > 3) and Sally [43] [44] (for n = 2); see also [46]. Intertwining 
operators such as L s have also been studied for semisimple groups of higher rank, see |27i 
[5S|. 

3.B. Invariant quadratic forms and hyperbolic spaces. As the reader will soon 
see, the natural parameter s for the spherical principal series is cumbersome to deal with 
geometric quantities. We shall freely use the following change of variable: 

Notation 3.4. From now on, we write t = (n — l)(s — 5). 

Let 

OO 

L 2 (<9H") = 0# fc 

k=0 

be the decomposition of L 2 (<9H") into A'-irrcducible components. Recall that, in the 
Klein model for H", where dH n = S n_1 is identified with the unit sphere in R", the 
space H k is simply the space of (restrictions to the sphere of) harmonic homogeneous 
polynomials of degree k; recall that for k > 2, we have 

/ n + k-l\ /n + fc-3 N 
n — 1 / V n — 1 



(3.ii) p k := dimff 



k 



Remark 3.5. When n = 2, it is more customary to decompose L 2 (9H 2 ) into irreducible 
components for the group SO(2); in this case, each space H k (k > 0) breaks into two 
invariant lines for SO(2). However, since we consider the action of the whole group 
K = 0{n), the spaces H k are irreducible, even in the case n = 2. 

Since the action of tt s (K) on L 2 (S"~ 1 ) does not depend on s and since L s commutes 
with this action, L s must preserve each of the subspaces H k . Hence the restriction of L s to 
H k is the multiplication by a scalar which we will denote by Afc(s). Up to a normalization, 
we have Ao(s) = 1 and for k > 1: 

(3.in) Ms) = ff i+nf rS n ~? = n — ■ 

For a proof see [5S] for n > 3 and [33] or [T3] for n = 2. From this formula one can easily 
compute the signature of (•, -) s . We write Sj = —^r + i for the parameter s corresponding 
to integer values j > of t. When s £ (0, So), the bilinear form (■, -) s is positive definite. 
When s > so we have: 

• if s G (sj, s j+ i) with j odd, (•, -) s has index T,i<k<j;koddPkl 

• if s G (sj,s j+ i) with j even, (■, -) s has index Eo<fc<j ; fcevenP*- 

Therefore, by (|3.iip . the possible values of the index of the form (•, -) s , when s > Sq, are 
exactly the numbers ( n ~j\ J ) for j > 0. 
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Example 3.6. When n = 2, the index can be any positive integer. 

Example 3.7. When n = 3, the possible indices are exactly the triangular numbers: 
(j + l)(j + 2)/2 (withj>0). 

For s € (sq,Si), the quadratic form has index 1 and the number t ranges over the 
interval (0, 1); this is the parameter which appears in Theorem [B] We will now use both 
the parameters t and s and always assume that s € (s , Si). Observe that the coefficients 
Afc(s) are real (since s is real), hence the quadratic form (■, -) s is defined over R. We can 
thus consider the subspace 

L 2 (dH",R) c L 2 (SH") 

of real- valued functions endowed with the form (-, -) s . Observe however that the space 
®k>\H k endowed with the scalar product — (-,-) s is not complete. This follows from 
the fact that the coefficients Xk(s) tend to as goes to infinity. Hence the pair 
(L 2 (<9H", R), (•, -) s ) is not isomorphic to a pair (Jt?,B) as described in the introduc- 
tion. However, if V denotes the completion of the space (Bk>\H k with respect to — (•, -) s , 
the group Isom(H") still acts on H° © V, preserving the extension of the form (•, ') s to 
this space. See for instance P3], Section 2.1, for more details. The need to complete the 
space L 2 (9H",R) will be illustrated in Proposition 13. Ill 

From now on, we will denote by the hyperbolic space associated to the pair 
(H° © V, (•,•)*) an d by g t : Isom(H") — > Isom(H^°) the corresponding representation. 
The linear representation ir s being irreducible (for s <G (so,si)), the representation gt is 
non-elementary and has no non-trivial, closed, totally geodesic invariant subspace. 

Let * be the point of H£° associated to the constant function 1. We record the following 
elementary observation: 

Lemma 3.8. The point * is the unique point o/H^° fixed by SO(n) = K° (a fortiori by 
0(n) = K). Hence, the map f t : H" — > defined by 

ft(9-o)=ir s (g)(*), G?eIsom(H")) 
is the unique Isom(H") -equivariant map from H" to H^°. 

Proof. There is only one X°-invariant line in L 2 (<9H") by transitivity of the i^°-action 
on 9H". This is still true in the completion H° © V since it is isomorphic as a JT-modulc 
to L 2 (<9H"). □ 

Note that the map 

(X,v,t) i — > f t (g\,v,id ■ o) 
is smooth in all its variables simultaneously. Note also that the map ft is harmonic (see 
for instance [JO] for the definition): if the tension field r(f t ) of f t was non-zero, it would 
be non-zero at every point and the geodesic tangent to the vector r(/ t )(o) at o would be 
(pointwise) K- invariant, contradicting the fact that K has a unique fixed point in H^. 

Let g u = g e ",o,id- We want to evaluate the distance between the point 

K s (gu)(*) = ft(g u ■ o) 

and * = ft(o) in H^°. This distance is given by the formula: 

cosh(d(7r s ( 5u ) (*),*)) = / Jac(g~ l )? +a db 
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where db stands for the normalized if-invariant volume form on dH n . We will denote by 
I u the integral above. The formula (|3.ip for the Jacobian yields 

(3.iv) Jac(g- 1 )(6) = (cosh(u) - b x sinh(u))~ (ri_1) (b £ <9H") 

where bi stands for the first coordinate of b £ S n_1 C R™ under the identification of dH n 
with the unit sphere S™" 1 via the Klein model. Recalling that t = (n — l)(s — 5), we 
obtain: 

(3.v) I u := cosh(d(7r s ( 3u )(*), *)) = / (cosh(u) + b x sinh(u))- {n - 1+t) db. 



Let us mention that in [14) . where the representations Qt were studied in the case n = 2, 
the following formula was established: 

1 f 2lT d-d 
cosh(d(g t (a u ) (*),*) = — 



2tt J \sh(u)e M + c1i(m)| 2 (*+ 1 ) 
where a u is the following element of SU(1, 1) C Isom(H 2 ) 

ch(it) sh(w) 
sh(tt) ch(u) 

This is coherent with Equation (|3.v[) : indeed a straightforward computation shows that 
the integral above equals: 

1 f 2 * d-d 



2ir J |ch(2u) + sh(2u) cos(i?)| 1+t ' 

Remembering that a u has translation length 2\u\ in H 2 , one recovers a formula of the 
same form as (|3.v[) . 

3.C. Geometric properties of the representations g t . To complete the proof of 
Theorem IBI and to prove Theorem [Cl we need to obtain estimates on the quantity /„. We 
first deal with its behaviour when u goes to zero. We will prove the following proposition 
(already established in [TJ] when n = 2). 

Proposition 3.9. We have, as u goes to zero: 



d(*M(*), *) = J t{t + H 1] ■ u + 0(u 2 ). 
V n 

We remark that it is not granted a priori that the diplacement of * under g u should 
have an analytic expression in u in infinite-dimensional spaces such as H°°; indeed this 
fails completely for actions on Hilbert spaces. 

Proof. The expression (cosh(it) + b\ sinh(u)) admits a Taylor expansion at u = 0, 

which to order 3 is 

(t + n-1) n (t + n-l)(t + n), , ^, 
\-{t + n- 1)M - ^— -u 2 + ±— 9 A ' b\u 2 + 0{u 3 ). 

Thus, after integrating, 

, , , w s , (t + n-1) o (t + n-l)(t + n) 9 , 
(3.vi) cosh(d(7r s ( 5u )(*), *) = 1 - ^ } -u 2 + ^ ^ >-u 2 c n + 0(u 3 ), 

where c n is the average of the function b\ on the sphere, which is c„ = 1/n by an easy 
symmetry argument. On the other hand, we can write d(Tr s (g u )(*), *) = \u + 0{u 2 ) for 
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some constant A, as follows e.g. from the existence of the expansion (|3.vi[) via the implicit 
function theorem. This implies that 

cosh(d(n s (g u )(*), *) = 1 + \\ 2 u 2 + 0(u 3 ). 

By identification in the two expansions for cosh(d(7r s (g u )(*), *)) we get A 2 = t(t+n—l)/n. 
This leads to the expression announced in the statement of the proposition. □ 

Since the two metrics ghyp and ft ghyp on H™ are proportional, the previous proposition 
implies that ft ghyp = +n ~^ 9hyp- I n particular, the curvature of H n endowed with the 
metric f I ghyp is equal to - t ( t +n-i) - 

Proposition 3.10. For n and t fixed, there is a constant k > such that 

Ke tu < I u < e tu 

for all u > 0. 

Proof. We work with the expression (|3.v[) for I u and begin with the upper bound. Since 
the term 

(3.vii) cosh(w) + b\ sinh(it) = e~" + (b 1 + 1) sinh(w) 

is > e~", it suffices to prove that 

(cosh(u) + bi sinh(u))- ( "- 1) (i6 



'S"- 1 

remains bounded by one. But the above expression is exactly one since it is the integral 
of the Jacobian of g~ x . For the lower bound, we shall only integrate (|3.v[) over —l<bi < 
e -2u _ 2_. In that region, the identity (|3.vii[) yields 

3 

cosh(u) + b\ sinh(w) < 

and thus the integrand is at least a contant times e ll (™- 1 +*). An elementary computation 
shows that the cap of S n_1 defined by b\ < e~ 2u — 1 has volume larger than rzrr e- • 
The lower bound follows. □ 

Note that Proposition 13.101 immediately gives us the end of the proof of Theorem [Bl 
Since gt preserves the type (Proposition ^. ip and since all hyperbolic elements of Isom(H") 
are conjugate to an element of the 1-parameter subgroup {g u }, the functions £ Bt and ^H" 
are proportional. According to Proposition 13.101 the proportionality constant is equal to 
t. We now collect all the previous informations to obtain a proof of Theorem [Cl 

Proof of Theorem^ We have already seen that there is an Isom(H")-cquivariant map 
H n — > which satisfies ft ghyp = * ( ' t+ ^~ 1 ' > ghyp (and is unique). We have also seen 
that ft is harmonic. Since the metric ft ghyp is proportional to the hyperbolic metric 
on H™, the harmonicity of ft is equivalent to the fact that the image of ft is a minimal 
submanifold, thus justifying the claim from Theorem IO (2). 

We now have to establish the first and third point in Theorem [Cl Let x and y in H" 
and denote by u the distance from x to y. There exists and element h £ Isom(H n ) such 
that x = h(p) and y = hg u (o) where g u = g e u ,o,id as in the previous paragraph. Hence 
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d := d H °°(ft(x),ft(y)) = dn°°{ft(9u(o)),ft(o)) = cosh 1 (I U ) where I u is the integral 
from the previous paragraph. Since \e d < I u < e d , we deduce from Proposition 13 . 101 that 

log(/c) + td H Ax,y) < d H ?(ft(x),ft(y)) < tdwfay) +log(2). 
This shows the first point. 

For the last point, we recall that quasi- isometric maps between geodesic Gromov- 
hyperbolic spaces extend continuously to the boundary. This concludes the proof since 
the estimates of point (1) are stronger than quasi-isometric. □ 

Knowing the translation length of a hyperbolic element g of Isom(H") in the rep- 
resentation g t , we can now prove that the fixed points of Qt(g) in <9H°° do not lie in 
L 2 (<9H n ,R). 

Proposition 3.11. Let g € Isom(H n ) be a non-trivial hyperbolic element of translation 
length u > 0. Let f G H° Q)V be an istropic vector representing the attracting fixed point 
of Qt{g) in dH^ . Then f does not lie in the dense subspace L 2 (<9H™, R) C H © V . 

Proof. We can assume that g — g u for some u. Note that if £ is an isotropic vector 
representing the attracting fixed point for Qt(g u ) one has Qt(gu)(0 = e*"£. Assume that 
there exists a non-zero function / € L 2 (i9H n ,R) such that: 

Up to replacing / by |/|, we can assume that / > 0. We rewrite the above equation in 
the following way: 

e tu f{b) = Jac( 5 - 1 )^T(&)Jac( g - 1 )(6)/fc 1 (6)). 

Using formula (|3.ivj) for the Jacobian, one sees that the maximum value of Jac(g~ 1 ) on 
<9H" is equal to e (" -1 )", we thus obtain: 

e ta /(&) = ^ C {gZ l ) s+ Hb)f{g-\b)) < e'-Jac^X&m.g- 1 ^)). 

Being in L 2 (9H",R), the function / is in particular integrable; we can thus integrate 
each side of the above equation. Since the two integrals are equal we must have: 

Jac^r+^/Cg,- 1 ^)) = ^"Jac^- 1 )^)/^)) 

almost everywhere. If / is non-zero, this implies that the following identity holds on a 
set of positive measure: 

Jac^- 1 )^)^ 4 ". 

This is a contradiction. □ 

Remark 3.12. Ln this text, we chose to deal with the full isometry group Isom(H ra ) of 
the hyperbolic space H". Note however that the actions gt are still non- elementary when 
restricted to the identity component Isom(H™)° o/Isom(H"). This follows for instance 
from the fact that an action of a group G on H°° is elementary if and only if it preserves 
a finite set in H°° U <9H°°. Also, it is easy to see that our proof that the g t are the only 
irreducible actions o/Isom(H") on H°° still applies to the group Isom(H™)°. 

4. Convex hulls and exotic spaces 

This section is devoted to the surprising properties of the spaces Ct described in The- 
orem [D] and Proposition El 



AN EXOTIC DEFORMATION OF THE HYPERBOLIC SPACE 



21 



4. A. The spaces C t - Let Ct be the closed convex hull of / t (H") inside H°°. We have: 

Lemma 4.1. The space Ct is minimal, i.e. it admits no nontrivial Isom(H") -invariant 
closed convex subset. Moreover, this is the unique minimal Isom(H n ) -invariant closed 
convex subset of H°° . 

Actually, it is easy to deduce from the results in |32j that any group G acting isometri- 
cally on a CAT(— 1) space X and having no fixed point in X U dX has a unique minimal 
closed G- invariant convex subset in X. However, in our case the fact that the compact 
subgroup K C Isom(H" ) has a unique fixed point in H°° makes the proof immediate. 

Proof of Lemma \4-. 1\ It is enough to prove that if C is a closed Isom(H")-invariant convex 
subset of H°°, then C contains Ct- But if C is such a set, the compact group K must 
fix a point in C. Since * is the unique if -fixed point in H°°, we have * G C. Hence the 
Isom(H")-orbit of * and its closed convex hull are contained in C, i.e. Ct C C. □ 

Theorem P|3|) yields a continuous Isom(H")-cquivariant map dH n —> dH°°. Its image 
is a non-empty Isom(H")-invariant subset / f (<9H") which is compact for the cone topol- 
ogy. We now consider the Klein model: that is, the space H°° is identified with the unit 
ball B in a Hilbcrt space and hyperbolic geodesies correspond simply to chords in that 
ball. Moreover, the topology on H°° corresponds to the norm topology on the closed ball 



Denote by C C H°° the closed convex hull of / t (9H"), which is by definition the inter- 
section of all closed convex subsets of H°° whose boundary at infinity contains ft(dH n ). 
The above discussion shows that C' is identified with the intersection of B and the 
closed convex hull (in the affine sense) of / t (9H") in B. It also makes it apparent that 
dC' = ft(diL n ). The Mazur compactness theorem [30] (an English reference is e.g. [31] 
2.8.15]) shows that the closed convex hull of f t (dH n ) in B is norm-compact. Therefore, it 
follows that C' is locally compact and closed in B. In conclusion, C' is a closed subspace 
of H°° and is proper as a metric space. 

Lemma 4.2. We have C' — Ct- In particular, Ct is a proper metric space in view of the 
above application of Mazur 's compactness theorem. 

Proof. By definition, Ct is the intersection of all closed convex subsets of H°° that contain 
/t(H"). Since the boundary of any such set contains ft(dH n ), we have C' C Ct- By 
minimality of Ct (Lemma 14. 1)) we have C' = Ct- □ 

Proposition 4.3. The Isom(H") -action on Ct is cocompact. 

Proof. Suppose for a contradiction that Ct contains a sequence Xj such that the distance 
d(xj, Isom(H™)a;o) to the orbit of xq = jt(o) tends to infinity. Upon replacing Xj with an 
Isom(H")-translatc, we can suppose that d(x 3 ■, Isom(H n )a;o) = d(xj,x ). Moreover, we 
can assume that Xj converges to some point £ € dC t = ft(dH n ). Choose a sequence j/j 
in H" such that ft{yi) converges also to £. Thus the Gromov product 



tends to infinity as i,j — > oo. Let zjj be the point of the geodesic [xo,Xj] at distance 
Li t j of xq. By construction, the distance from z.ij to the orbit of xq tends to infinity. By 
Gromov-hyperbolicity, there is a constant bounding the distance from Zjj to the geodesic 



B. 
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[xo, ft(yi)]- Theorem Q[T|) shows that the image under f t of the geodesic [o, yi] is a quasi- 
geodesic controled by constants depending only on t. Thus, by Gromov-hypcrbolicity, 
a similar constant bounds the distance from [xq, ft{yi)] to /t(H"). Putting everything 
together, Zij remains at bounded distance of f t (H. n ), which is none other than the orbit 
of Xq; this is a contradiction. □ 

Proposition 4.4. The representation Qt induces an isomorphism Isom(H") = Isom(Ct). 

Proof. For simplicity, let G = Isom(Ct) and L = Isom(H"). Since Qt induces a proper 
homomorphism, we consider I as a closed subgroup of the locally compact group G (for 
the compact-open topology, recalling that C t is proper). Moreover, L is cocompact in G 
because it acts cocompactly on Ct- Since L is almost connected (L° has index two in L), 
the connected component G° is cocompact in G. Therefore, Yamabe's theorem [331 4.6] 
implies that G is a Lie group after possibly factoring out a compact normal subgroup 
K < G. However, G has no non-trivial compact normal subgroup because the fixed- 
point set is a non-empty closed convex G-invariant set, which implies = Ct by 
minimality of Ct (even amongst L- invariant such sets). 

Thus, we have a connected Lie group G° of finite index in G (and containing L°). 
The amenable radical of G° is trivial (see e.g. [5J 1-10]), so that G° is a semi-simple Lie 
group without compact factors. Since L° is closed cocompact in G°, this forces G° = L° 
(e.g. using the Karpelevich-Mostow theorem). At this point, we have realized G as an 
extension 

1 — > L° — > G — > F — >■ 1 

with F finite. We know from Lie theory that the group of outer automorphisms Out(i°) is 
of order two, with the non-trivial element being implemented by conjugation by orientation- 
reversing elements of L. The above group extension induces a representation G — > F — > 
Out(L°) whose kernel is L° .Zq(L°), where Zg(L°) is the centraliser of L° . The central- 
izer has to be trivial by minimality of Ct, see e.g. [5J 1.10]. Therefore, L° is of order two 
in G and thus G = L, finishing the proof. □ 

Corollary 4.5. Ift,t' G (0, 1] are distinct, then the spaces Ct and Ct' are not isometric, 
even upon reseating the metric. 

Proof. Suppose that there is some scalar /i > and a bijection h: C t — > C t > such that 
d(h(x),h(y)) = [id(x, y) for all C t . The conjugation of Q t > by h yields a continuous 

homomorphism q\, : Isom(H") — > Isom(Ct). In view of Proposition 14.41 the maps q\, and 
Qt are isomorphisms onto Isom(C4) and differ by an automorphism of L = Isom(H"). The 
description of the outer automorphisms of L° given in the proof of Proposition l4.4l implies 
that the group L = Isom(H") has trivial outer automorphism group upon considering the 
natural map from Out(L) to Out(L°). In conclusion, the two actions on Ct are conjugated 
by an element of Isom(H" ) and thus h is Isom(H")-cquivariant after conjugation. 

We can now apply point (QJ of Theorem [Bl and deduce fit = t' . Moreover, since the 
subset ft (H") is intrisically characterised in terms of the (conjugacy class of) the action 
Qt on Ct by Lemma 13. 8[ the curvature given by Theorem 0|2|) must rescale according to 
fi~ 2 . Thus we have 

— n —n 
t'(t' + n - 1) ~ fi 2 t(t + n-l) 
which together with fit = t' implies t = t! , as desired. □ 
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At this point the proofs of Theorem [Dl and Proposition [E] arc complete. 

The local compactness of the exotic space Ct established in Proposition 14.31 could not 
have happened if the ambient space were a Hilbert space rather than H°°. as shown by 
the following fact. 

Lemma 4.6. Let G be a simple Lie group (or a simple algebraic group over a local field) . 
If G acts continuously by isometries on a Hilbert space V , then either it fixes a point or 
it does not preserve any non-empty closed locally compact convex subspace of V . 

Proof. We can assume that the linear part of the action has no G-invariant vectors. 
Indeed, since G has no non-zero additive characters, the G-invariant part splits off a 
summand preserved by the isometric action. Suppose for a contradiction that G preserves 
a non-empty closed locally compact convex subset G C V but has no global fixed point. In 
particular, G is non-compact and hence contains an element g generating an unbounded 
subgroup of G (e.g. a split torus element). Since G is a proper CAT(O) space, g fixes a 
point either in G or in its boundary. In the former case, all of G will have a fixed point by 
a Mautner type argument, which is absurd (we already recalled earlier that unbounded 
G-actions are proper). In the latter case, the linear part of the action of g fixes a vector. 
This implies that the linear G- representation has a fixed vector by Moore's theorem ( [34] ; 
see also [23[ 5.5]), contrary to our preliminary reduction step. □ 

4.B. Continuity of the family of actions. In this section we shall prove that the map 
(0,1] Bin- {Ct,*) is continuous for the pointed Gromov-Hausdorff topology. For an 
introduction to this topology, see [TJ gT| as well as [3] 1.5] and []j|J chap. 3]. It builds 
upon the Hausdorff distance defined in [22] (Kap. VIII §6, pp. 293 sqq). 

We shall in fact prove more: the spaces Ct together with their Isom(H") -actions vary 
continuously for the natural "strong" topology for actions on metric spaces, which is an 
cquivariant version of the pointed Gromov-Hausdorff topology. Regarding terminology, 
the reader should be warned that the "equivariant Gromov-Hausdorff topology" intro- 
duced by Paulin [41] is very different from the "strong" topology used below; for instance, 
Paulin's topology does not imply convergence of the underlying spaces. 

Recall that the coradius of a subset A of a metric space X is the (possibly infinite) 
number 

covadx(A) = sup inf d(x, a). 

x£X a ^ A 

From now on and all along Section [4.B1 we will assume that all metric spaces are geodesic. 
If A" is a metric space and x a point in X we wil denote by Bx(x,r) the closed ball of 
radius r around x in X. 

Let G be a topological group. A pointed metric G -space is a triple {X, *, it) where X is 
a metric space, * € X and 7r is a continuous G-action by isometries on X; we denote also 
by 7r: G — > Isom(X) the corresponding homomorphism and write often gx for n(g)(x). 

We define a topology on the class of all pointed metric G-spaces as follows. Given 
R > 0, e > and a compact subset M C G such that 1 <E M, let J/r^m^; *,7t) be the 
class of all pointed metric G-spaces (X', *', tt') such that there is an (M, e)-map f relative 
to the balls Bx>(*',R) and Bx(*,R). This shall mean, by definition, that / is a map 
/ : X' -> X such that 

(1) /(*')=*, 
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(2) the Hausdorff distance between f(Bx>(*',R)) and Bx(*,R) is < e, 

(3) \d(f(x),f(y)) - d(x,y)\ < e for all n' (M) {Bx 1 (*', R)) , 

(4) d(gf{x), f(gx)) < e for all x € B x >{*' , R) and all g g M. 

It is clear that these [/r. £ .m(A, *> tt) form a neighbourhood base. We shall refer to 
this as the topology of strong convergence of pointed metric G-spaces. This topology is 
very close to the one considered by Fukaya Q2]. Notice that when G is the trivial group, 
we essentially recover the pointed Gromov-Hausdorff convergence of metric spaces |19| . 
Observe also that, if G is cr-compact (e.g. locally compact and connected or second count- 
able), then every pointed metric G-space has a countable neighbourhood base. 

Before turning to the proof of the continuity of the map t € (0, 1] i-> (Gt, *, Qt) for 
the topology just defined, we prove a preliminary fact concerning isometries of CAT( — 1) 
spaces. We recall that a hyperbolic isometry g of a complete CAT(— 1) space X admits a 
unique axis L g C X. We shall denote by G dX its forward, respectively backward 

endpoints and by £x(g) the translation length of g. We recall, but shall not use, that for 
any x £ X one has g n x — > £^ as n — > ±oo. 

Proposition 4.7. Let X be a complete CAT(— 1) space and endow the set H of its 

hyperbolic isometries with the topology of pointwise convergence. Then the map 

H — > dX 
9 ^ C 

is continuous on each level-set of £x '■ H — > R+ . 

A more refined analysis reveals in fact that continuity holds whenever lx remains bounded 
away from zero, but we will not need this. 

We shall use the following fact from CAT(— 1) geometry. 

Lemma 4.8. Let h be a hyperbolic isometry of a complete CAT(— 1) space X . There is 
a continuous, increasing function c: [£x(h),oo) — > [0, oo) depending only on £x(h) with 
c(£x{h)) = and such that for any x G X we have d(x,Lh) < c(d(hx,x)). 

Proof of the lemma. Fir r > 0. Elementary geometry of the hyperbolic plane H 2 shows 
that there is a continuous, increasing function c: [r, oo) —> [0,oo) with c(r) = and 
satisfying the following property. If x,y,z,t are four points in H 2 with d(z,t) = r and 
with (hyperbolic) angles Zz(x,t) and Z^(y,z) both at least tt/2, then d(x, z) < c(d(x,y)) 
(and d(x, y) > r). In fact, we will use it only for the special case where d{x, z) = d(y,t). 

To prove the lemma, we set r = £x(h) and choose x, y,z,i £ H 2 to be a sub-embedding 
in the sense of [31 II. 1.10] for the following four points: x, y = hx, z the projection of x 
to Lh and t = hz, which is he projection of y to Lh- The definition of a sub-embedding 
is that one should have 

d(x,y) = d(x,y), d(z,t) = d(z,t), d(x, z) = d(x,z), d(y,t) = d(y,t) 

and 

d(x, t) < d(x, t), d(y, z) < d(y, z). 

In particular, since the Alexandrov angles Z. z (x,t) and Z t (y,z) are > tt/2 by a property 
of projections, we conclude that the points x,y,z,i are as required above; the lemma 
follows. □ 
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Proof of Proposition ^. 7\ Fix g G H and p g G Lg. For any ft, G -ff, wc shall denote by 
the projection of p g to Lh\ thus /i"p^ travels on Lh towards £t ■ Fix some D > 0. In view 
of the definition of the cone topology on dX, it suffices to prove the following claim: for 
any n G N, there is a neighbourhood U oi g in its level-set in H such that d(p gi ph) and 
d(g n p g ,h n ph) both are at most 13. 

The function ft i— > d(hp g ,p g ) is continuous; therefore, Lemma 14.81 shows that we can 
find U such that d(p g ,ph) < D/2 for all ft G U. We further restrict U by requiring 
d{g{g^Pg), h(gip g )) < D/2n for all j = 0, . . . , n — 1. This implies 

d(h n - V/V < ^ Vj = 0,...,n-1 

and hence d(g n p g ,h n p g ) < D/2. Wc conclude d(g n p g ,h n ph) < D and the claim is 
proved. □ 

Given a subset i C X of a geodesic metric space X, let [A] be its closed convex hull. 
We recall that X is non- positively curved in the sense of Busemann [6] if the distance 
function is convex (in particular the space is uniquely geodesic); see [3j ILL 18]. 

Lemma 4.9. Suppose that X is non-positively curved in the sense of Busemann. Then 
the operation A i— >• [A] is 1-Lipschitz for the Hausdorff distance on bounded non-empty 
sets AC X. 

Proof. For any subset A, let A' be the collection of midpoints of all pairs of points in A. 
We thus obtain a sequence A^ = A, A^ n+1 ^ = (iW)' D A (n \ Moreover, [A] coincides 
with the closure of the union of all A^ n > . Therefore, it suffices to show that the operation 
A i— > A 1 is 1-Lipschitz for the Hausdorff distance on bounded non-empty sets. This follows 
from the definition of convexity. □ 

In this paragraph we will be constantly using both the parameter t and the parameter 
s. We thus remind the reader that they are related by the equation t = (n — l)(s — 5). 
To keep track of the parameter t when dealing with the various bilinear forms we have at 
hand, we will denote by B t the symmetric bilinear form (•, -) s introduced in Section [3.AI 
Finally, we will denote by | ■ | the L 2 -norm on L 2 (<9H") and if v G L 2 (<9H") by Vk the 
component of v in the subspace H k C L 2 (<9H"). 

To study the continuity at t = 1, we will need the following proposition. Recall that 
the special values Sj of the parameter s were defined by Sj = + i ; they correspond 
to the value j of the parameter t. 

Proposition 4.10. The representation tt s1 preserves the space V2 ■= ©^L 2 ^ k as well 
as the inner product U2 on V2 defined by: 

There is of course a similar result replacing s± by sq or by Sj for some j greater or 
equal to 2, see [2"5] . 

Proof. We already know that the form B t is invariant by n s for all s > 0, even if tt s is 
reducible. Hence tt Si must preserve the kernel of the form B±, which is nothing else than 
the space V%. Using Equation (|3.iii|) . one proves easily that the bilinear form U2 on V2 
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defined by the limit above satisfies: 

fe-i 



U 2 (v,v) = — -V TT 



3 ~ 1 1 ,2 



n(n + 1) H / + n 

v y fc>2 j=2 J 

We now show that it is invariant by the (restriction of) the representation 7r Sl . Let v € V 2 
and g G Isom(H™); write 

7r s (g)(v) = a(t) + c(t) 

where a(t) G H° Q) H 1 and c(t) G l/ 2 . Note that a(t) and c(i) n Sl (g)(v) as t ->• 1. 
Since 7r s (<7) preserves -Bt, we have: 

--Bt(«,«) _ -B t (o(t),o(t)) -fl t (c(t),c(t)) 
1 -f 1 - i 1 

Observe that the forms -B t satisfy (for t G (0, 1) and v G L 2 (<9H")): 

(4-i) < cm 2 . 

From this one easily deduces that the first term above converges to and the second term 
to U 2 (ir Sl (g)(v),Tr sl (g)(v)). □ 

Each operator ir Sl : L 2 (9H") — s- L 2 (<9H") can be represented by a matrix associated 
to the decomposition 

L 2 (<9H") = (H°@ H 1 )® V 2 

which has the following form: 

A{g) 
C{g) D(g) 

From now on, we will denote by 7T Sl the representation of Isom(H n ) on L 2 (<9H") defined 
by ^1(5) = A{g) © D(g)\ it preserves the quadratic form of hyperbolic type U\ © — U 2 
where U\ is the restriction of B\ to H° © H 1 . 

Let ut ■ L 2 (<9H") -> L 2 (<9H") be the map defined by 

, , f v v G © tf 1 

Ut ^ = \VT—tv veV 2 

For each s G (sq, Si), we define a new representation tt s of Isom(H") on L 2 (<9H") by: 

TT s (g) := u t o ir s (g) o u^ 1 . 

To study the continuity of the family tt s at t = 1 we will need the following: 

Lemma 4.11. Let v G V 2 and g G Isom(H"). Write 

n s (g)(v)=a{t) + c{t), 

where a(t) <E H° ® H 1 and c(t) G V 2 . Then \a(t)\ = 0(1 - t) as t goes to 1. 

Proof. Since the form B t , restricted to H°(BH l , converges to a non-degenerate symmetric 
bilinear form as t converges to 1. it is enough to check that 

B t {a{t),u) = 0{l-t) 

as t goes to 1, for each fixed u G H° © H 1 . But B t (a(t),u) = B t {v s {g~ 1 )(u)) which 
is bounded by (1 — t)\v\ ■ |7r s (<7 _1 )(it)| thanks to (|4~i)) . Since n s {g~ 1 ){u) converges, this 
concludes the proof. □ 
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The next proposition shows that, up to a renormalization, one can include the family 
of actions (gt)te(o,i) m t° a continuous family of actions on H°° defined on (0, 1]. Its proof 
uses Lemma \A. Ill and will be left to the reader. 

Proposition 4.12. As s — > Sx (equivalently, as t — > 1) the representations n s converge 
to tt s1 : for any v £ L 2 (<9H") and g £ lsom(H™), Tr s (g)(v) —> Tr si (g)(v). 

The representation tt s preserves the symmetric bilinear form B t defined by: 

B t (v,v)=B t (ut 1 (v),Ut 1 (v)) ift£ (0,1), 

B 1 = U l ® -U 2 . 

Let B be the bilinear form on L 2 (9H") defined by B(v,v) = |v | 2 — J2T=i \ v k\ 2 - For 
t £ (0, 1], we choose a X-cquivariant operator c/j* of L 2 (<9H") such that 

B(<p t (v),(p t (v))=B t (v,v) (v £ L 2 (<9H™)) 

(</?' is unique up to a choice of sign on each H k ). From now on, we will denote by H°° 
the hyperbolic space constructed from the form B and by Qt (t £ (0, 1]) the action on 
H°° induced by the linear representation 

ip* o7r s o ((/j*)" 1 . 

Let it : H™ — > H°° be the unique gt-equivariant map from H™ to H°°. Note that 
it = (p* out /* f° r t £ (0, 1). We will make a slight abuse of notation and will still denote 
by Ct the smallest invariant convex set for g t and by * £ Ct the unique fixed point of the 
group K = 0(n). Note that for t — 1, Ct is nothing else than an n-dimcnsional totally 
geodesic subspacc of H°° . 

Lemma 4.13. The map 

(0, 1] x dU n 

{t,x) 

is continuous. 

Proof. Since, for a fixed t, the image of it is equal to the orbit of a point of H°° under 
Qt(K), it is enough to prove that for a fixed £ £ <9H™, the map t i— > it{£) is continuous. 
Choose a one-parameter semi-group {.g r }r>o of hyperbolic elements of Isom(H") whose 
attracting fixed point in <9H" is £, normalized by £nn(g r ) = r. Then Qt(9i/t) is continuous 
in t £ (0, 1] and has constant translation length in H°°. Since it(£) is the attracting fixed 
point of Qt{gi/t)i Proposition 14. 71 ensures that 1 1— > is continuous. □ 

We arc finally ready for the main result of this subsection. 

Proposition 4.14. The map (0, 1] 9 t i— > Ct is continuous for the strong topology on 
pointed metric Isom(H") -spaces. 

Proof. Fix to £ (0, 1], R > 0, e > and a non-empty compact subset M of Isom(H"). Let 
D > be large enough so that g t {M)(B a ^ (*, R)) C B H °= (*, R + D) whenever |t— *o| < e- 
We work in the Klein model for H°° , hence H°° U 9H°° is identified with the closed unit 
ball of a Hilbert space Jt? in such a way that * coincides with the origin of Jtff. Notice that 
hyperbolic balls centered at * correspond to Euclidean ball centered at the origin (and 
of radius < 1). Moreover, the hyperbolic and Euclidean distances on Bn°° (*, R + D) are 
bi-Lipschitz to each other, so that we can argue with the Euclidean one for the remainder 
of the proof. 
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For any t G (0, 1] we denote by /: C\ — >• Ct the nearest point projection (in 34?). In 
view of the continuity of Qt (g) in t and g, the proposition will follow if we prove that when 
t is close enough to to, the Hausdorff distance between Bc t (*,R + D) and Bc tQ (*, R + D) 
is less than, say, e/2. Therefore, it suffices to establish the continuity of the map t H> C\ 
when C t is considered as a bounded subset of 34? \ equivalently, we can replace C t by its 
closure C t . At this point we recall that C t can be seen as the (affine) closed convex hull 
of it(d~H n ), see Section T4.AI Since <9H™ is compact, Lemma [4.131 implies that the map 
t i y it(dH n ) is continuous on (0, 1] for the Hausdorff topology on subsets of the Hilbert 
sphere. Therefore we are done by Lemma l4~9l □ 

Recall that the action tt is called (metrically) proper if the function d(ir(g)x, x) tends 
to infinity as g leaves compact subsets of G (for some, equivalently any, x G X). 

Proposition 4.15. The quantity coradx(7i"(G)(*)) is upper semi- continuous at (X,*,tt) 
on the class of all pointed geodesic metric G-spaces if tt is proper. 

Proof. Let (X, *, tt) and 5 > be given; write r = coradx(7r(G)(*)), which we can assume 
finite since otherwise the proposition is void. We choose < e < 5 / 4, R > 2r + 6e and a 
compact set MCGso large that g* G Bx(*,R) implies g G M. This is possible since tt 
is proper. 

Suppose for a contradiction that (X' ,tt') is geodesic and is in Ur^m(X,*,tt) but 
contains a point y with d(Tr' (G)(*'),y) > r+S. Upon translating, we can moreover assume 
d(*' , y) < d(7t' (G)(*'),y) + e. Let z be the point at distance r + 4e from *' on a geodesic 
connecting *' to y; this point exists since 4e < 5. 

Let /: X' — > X be as in the definition of Ur .e.m(X, *, tt). Since d(z, *') = r + 4e, 
the point z is in B X >{*',R) and we have d(f(z),*) < r + 5e. Let g G G be such that 
d(g*,f(z)) < r + e. Notice that g* belongs to the ball Bx(* 7 R) by the choice of R; it 
follows that g G M. Now we have 

d(g*',z) < d(f(g*'),f(z)) + e < d(gf(*'),f(z)) + 2e 

= d(g*,f(z)) + 2e < r + 3e. 

It follows 

d{g*',y) < d(y,z) + d(g*',z) < d(y,z)+r + 3e 

= d(*',y)-d(*',z) + r + 3e = d(*',y)-e < d(/(G){*'),y), 
which is impossible. This contradiction finishes the proof. □ 

As a consequence of this Proposition, we have that coradc t (it (H™)) converges to as 
t converges to 1, as announced in Theorem iFl 

4.C. Towards a renormalization limit. In this paragraph we make a few remarks 
concerning the behavior of the spaces Ct and of the actions Qt as t converges to 0. The 
next proposition shows that the right scale to renormalize the family Ct as t goes to is 

Vt. 

Proposition 4.16. Fix any g G Isom(H"). Then, as t G (0,1) converges to zero, the 
ratio 
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Vt 

remains bounded, and bonded away from zero unless g is in 0(n). 

We start with an elementary observation. 

Lemma 4.17. Let g be a smooth diffeomorphism of a compact Riemannian manifold M 
and n > 1 . We endow M with its normalized volume form. If g does not preserve the 
Riemannian measure of M , then the function 



t.— > / |Jac(<?)|^^ 
Jm 

has a Taylor series around t = starting with 1 + ait + ■ ■ ■ where ai > 0. 

Proof of the lemma. We only need to show that the derivative at t — is positive, recall- 
ing J M |Jac(<7)| = 1. Differentiating under the integral sign, that derivative is 



n 



l -( |Jac( 5 )|log(|Jac(.g)|) = l — f log (| Jac^" 1 )!) , 

1 Jm n ~ 1 Jm 

where the second expression is obtained by change of variable. The equality case of 
Jensen's inequality shows that the latter expression is positive unless |Jac(g~ 1 )| is iden- 
tically 1. □ 

In other words, a\ measures the Kullback-Leibler divergence between |Jac(<?)| and 1. 

Proof of Proposition \4-16] By the polar decomposition, any element g can be written 
.9 = kg u k' where k, k' £ O(n) and g u = (? e u ,o,id as before. Thus it suffices to consider the 
case of g u (and we can assume u > 0). The formulas (|3.iv[) and p.vp of Section [3] give 



cosh d(7r s (g„ )(*),*) = / Jac(g u ) »-i . 

Therefore, Lemma [4.171 combined with the Taylor series of the hyperbolic cosine concludes 
the proof. □ 

The representation ir So on L 2 (<9H") was defined by n So (g) ■ f = | Jac(.g _1 )|/ o g~ x . It 
preserves the subspace V\ C L 2 (5H") of functions with mean as well as the inner 
product Ui defined by: 

TT I \ V - B t( V > V ) I f~- \T \ 

Ui(v,v) = hm . (v £ Vi) 

t^o t 

The map c : Isom(H") — > Vi defined by 

c(s)=7r. (ff)(l)-l, 

(where 1 stands for the constant function equal to 1) is a cocycle, since it is a formal 
coboundary. Let g be the associated affine action: 

Qo{g)(v) = n So (g)(v) + c(g). 

Note that this action is considered in [TUl chap. 3, § 3.4]. However, there the authors are 
interested in the relation between the family of unitary representations n s for s £ (0, 5) 
(the complementary series) and 7Ti . Here, we approach 7r 1 from the other side of the 
looking glass. It would be interesting to determine whether the Isom(H")-spaces ^jCt 
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admit a limit as t goes to and if they do, to relate the limit to the affinc action go just 
described. 

A related problem is to determine minimal invariant convex sets for affinc actions on 
Hilbert spaces. It is asked in [T3] (see Remark 4.9 there) whether there exists an isometric 
action of a semisimple Lie group on a (non-zero) Hilbert space Jf? for which the closed 
convex hull of any orbit is equal to all of Jff. In that context, we propose the following 
dichotomy. 

Proposition 4.18. Let G be a topological group with a continuous isometric action on a 
Hilbert space J4f. Assume that its linear part is irreducible and without K -invariants for 
some compact subgroup K . Then any closed G -invariant convex set either is Jti? or has 
empty geometric boundary. 

Recall that a Gelfand pair (G, K) consists of a locally compact group G with a compact 
subgroup K such that the convolution algebra si of compactly supported bi- if -invariant 
continuous functions on G is commutative. This algebra is called the Heche algebra. 
Examples include all simple connected center-free Lie groups G with a maximal compact 
subgroup K; the criterion originally due to Gelfand also applies to G = Isom(H") and 
K = O(n), see e.g. [17]. 

Corollary 4.19. Let (G,K) be a Gelfand pair with G compactly generated. Consider a 
continuous isometric action of G on a Hilbert space J^f with irreducible linear part. If 
G contains an element with positive displacement length, then Jif is the only non-empty 
G-invariant closed convex set. 

Proof of the corollary. Let 7r be the linear part of a, b the cocycle and jgGan element 
with positive displacement length. The existence of g implies that G has no fixed point, 
i.e. the action is non-trivial in cohomology. By [T^l Prop. V.3] this implies that ir(K) 
has no invariant vectors. Let C be a closed convex invariant set. If C is non-empty, we 
can assume that <G C. In view of Proposition 14. 181 it suffices to prove that the segment 
[0, a(g n )(0)} converges to a ray in C. The assumption on g implies that \a{g n ){§)\ / n 
converges to a positive number. Since 

n-l 

a{g n W)^Y.^y{b{g)), 

j=o 

the von Neumann ergodic theorem implies that a(g n )(0)/n converges to a non-zero vector. 

□ 

Remark 4.20. The argument in the proof above, combined with Moore's theorem, implies 
that if G is a simple Lie group, all elements have displacement length for any isometric 
action of G on a Hilbert space. 

We will need the following lemma, which shows in particular that for compact groups, 
cyclic vectors are "positively cyclic" in the absence of invariant vectors. The assumption 
on invariant vectors is of course necessary. The compactness of the group is also necessary, 
as shown by the example of the regular representation of an infinite discrete group. 

Lemma 4.21. Let n be any continuous linear representation of a compact group K on 
a (Hausdorff) locally convex topological vector space V over R. If V = 0, then the 
closed convex cone generated by the K-orbit of any v € V coincides with the K-cyclic 
subrepresentation generated by v. 
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Proof of the lemma. It suffices to show that the closed convex cone generated by ir(K)(v) 
contains —v. Let thus U be an arbitrarily small convex neighbourhood of —v. There is 
an open neighbourhood A of the identity in K such that ir(cj)(v) £ — U for all g £ A. Let 
[i be the normalized Haar measure of K. We claim that 

(4-ii) -77T / ir(g){v) dfi(g) 

MA) Jk\a 

belongs to U , in which case the proof is complete. 

To prove the claim, observe that J K ir(g)(v) d^t(g) is if-invariant and hence vanishes. 
Therefore, the expression in (|4.iip coincides with — X4 7r (5)( u ) dfi(g). Thus the claim 
follows from the choice of A and the convexity of — U. □ 

Proof of Proposition \4- 18\ We denote by a the isometric action on Jf? and write 

a(g)(v) = n(g)(v) + c(g) 

for v £ Jtf. Consider a closed invariant convex set C. We must prove that if C contains 
a half-line, then C = Jff. Let c(t) = xq + tv (t > 0) be a half-line in C and assume by 
contradiction that C 7^ Jif. Then, there exists a non-zero linear form tp on Jf? and a 
constant a such that 

C C {x £ je,ip{x) > a}. 

Since c(i) is in C, we must have tp(v) > 0. We can apply the same result to any vector 
of the form ir(k)(v) for k £ K since C contains the ray 7r(fc)(c(i)). In particular we have 
tp(w(k)(v)) > for all k in K hence <f(u) > for any vector u which is a linear combination 
with positive coefficients of the vectors {ir(k)(v)}k£K- According to Proposition 14 . 2 1 1 we 
must have <p(—v) > as well, hence (f(v) = 0. Hence the geometric boundary of C is 
contained in the boundary of a proper closed linear subspace of M' '. This is impossible 
since ir is irreducible. Therefore C = J%?. □ 

5. Further results 

5. A. Complements on trees. One of the early motivations for this article was the 
analogy with representations of automorphism groups of trees. We recall in particular 
the following result from @], in which it is implicitly assumed that the tree has no leaf 
(and wc shall keep this as a standing assumption). 

For any simplicial tree 2T and every A > 1 there is a representation 
tt\: Aut(^*) — > Isom(H°°) and a ii\-equivariant map 3~ — > H°° ex- 
tending continuously to the boundary with 

coshd(* A (x),* A (y)) = A d ^ 

for any vertices x, y £ £? . Moreover, ^\{^) has finite coradius in the closed 
convex hull of^x{d^) in H°°. 

The argument of Section 14. Al allows us to show that for locally finite trees, the con- 
clusion can be strengthened. Just as for Isom(H n ), wc obtain "exotic" proper CAT(— 1) 
spaces: 

Proposition 5.1. If the simplicial tree S? is locally finite, then the closed convex hull C 
of ^ \(d£?) in H°° is locally compact. Thus, i/Aut(i^) acts cocompactly on 2? ', then (via 
tt\ ) it acts cocompacty on C. 
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Thus, for instance, we deduce that the finitely presented torsion- free simple groups 
constructed in [5] appear as "convex-cocompact" subgroups of isometrics of H°° x H°° . 

Proof of Provosition \5. 1\ The second statement follows from the first since has 
finite coradius in C. As for the first, it follows as in Section T4.AI by applying the Mazur 
compactness theorem in the Klein Model. □ 

By the result of [3] quoted above, the actions of Aut(i^) on H°° given by ir\ are 
non-elementary unless the Aut(^)-action on & is itself elementary. Thus, we can also 
consider the canonical convex subspace C\ C H°° provided by Lemma T4. II This subset 
is contained in the convex hull C of Proposition ISTTl for instance because Lemma B~T1 can 
be applied to both H°° and C . However, in contrast to Lemma 14.21 it seems that the 
minimal subset could be smaller than C in general. At any rate, even for regular trees, 
the initial representation constructed in [4] is far from irreducible before passing to a 
smaller subspace, and indeed there is an infinite-dimensional subspace of K-ftxed vectors, 
denoting by K the stabiliser of a vertex. 

5.B. Actions on the symmetric space of 0(p, oo). Let Jrf? be a separable real Hilbcrt 
space endowed with a basis (ej)j>x- In what follows, we will denote by 0(p, oo) the group 
of linear operators from Jf? to itself which preserve the symmetric bilinear form B p defined 

by 



Bp I ^ ^ XiCi J — X-^ -\- 



2 \ 2 



p L^i 

i>l i>l I i>p+l 

We also denote by X(p, oo) the space of all p-dimensional subspaces V C Jt? on which 
the form B p is positive definite. The space X(p, oo) can be endowed with an 0(p, oo)- 
invariant distance which turns it into an infinite dimensional symmetric space; see [15, 16 
for an introduction to this space. In this section, we collect a few remarks concerning 
isometric actions of Isom(H") on X(p, oo). 

As we have seen in Section [3] , the classical study of intertwiners for the spherical 
principal series provides a whole lot of irreducible representations 

Isom(H n ) — >0(p,oo), P=( n ~ 1 |' ? ) where j= 0,1,2,... 

Problem 5.2. Show that the above representations exhaust all possible irreducible repre- 
sentations Isom(H") — > 0(p, oo). 

This would in particular restrict the possible ranks p when n > 3, whilst we have al- 
ready observed that for n = 2, every rank p S N can occur. We propose the following first 
evidence for Problem 15.21 where we restrict to representations of the identity component 
Isom(H™)° of Isom(H"). 

Theorem 5.3. Let p be an integer with 2 < p < n. Then there is no irreducible repre- 
sentation Isom(H")° — > 0(p, oo). 

Of course, the first subquestion of Problem 15.21 left open by the theorem above is the 
existence of irreducible representations 

Isom(H™)° — S- 0(2, oo) 

when n > 3. 
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For unitary representations, the following proposition is well-known. It cannot possibly 
hold for arbitrary representations on a Hilbcrt space, but finite index sesquilinear forms 
provide just enough spectral theory to obtain the conclusion. 

Proposition 5.4. Let (G, K) be a Gelfand pair and k = R or C. Let n be a continuous k- 
linear representation of G on a k-Hilbert space Jif preserving a continuous, nondegenerate 
(sesqui)linear form of finite index. If n is irreducible, then the space of K -invariant 

vectors has k-dimension at most 1 if k — C and at most 2 if k = R. 

As always, irrcducibility means that there is no closed G-invariant proper subspace. 

Proof of Proposition \5.4\ We denote by srf the convolution algebra of compactly sup- 
ported bi-A'-invariant continuous functions on G. We consider the more complicated case 
k = R; it will be clear from the proof that it applies to k = C, with minor simplifications. 
Let B be the bilinear form under consideration. 

For any continuous linear operator T of Jrff , we denote by its -B-adjoint, i.e. the 
unique continuous linear operator such that B(Tu, v) = B(u, T^v) holds for all u, v G Ji?. 
The B-adjoint exists: it can be produced as = JT*J if J = J* = J -1 is the operator 
with B(x,y) = (Jx,y). By assumption, we have 

(5.i) 7r(p)t = TrCg- 1 ) V.g G G. 

We can extend ir to a representation of the convolution algebra C C (G) of all compactly 
supported continuous functions on G. Let (ik be the normalized Haar measure of K, 
seen as a measure on G. Then \ik is a convolution idempotent and 

fi K * C C (G) * [Ik = ^ = l-iK* * 

Further, we can define the continuous linear operator P := tt(pk), yielding a A-invariant 
operator P: Jtf? — > Jf? K which is the identity on Jf? K . In particular, the space Jf? K is 
■eZ-invariant. 

We claim that B is non-degenerate on J4? K , thus turning it itself into a quadratic space 
of finite index, say q > 0. Indeed, suppose that v is a non-zero element of . There is 
w G with B{v,w) 0. Using (|5.i[) . we find P^ — P and thus B(v,w) = B(Pv,w) = 
B{v, Pw), which proves the claim. 

Next, in view of the definition of and of the fact that a Gelfand pair is unimodular, 
we observe that (|5.ip also implies that srf is symmetric in the sense that a' € for all 
a G g/. A result of Naimark (Corollary 2 in [36]) states that a commutative symmetric 
algebra of operators of a complex Hilbert space endowed with a sesquilinear form of 
finite index q preserves a (non-negative) subspace of dimension q. Applying this to the 

£g> C-representation on the complexification endowed with Be, we deduce that 
srf preserves a finite-dimensional subspace of . At that point, the classical finite- 
dimensional Schur lemma applies and we find a subspace L C tf? K of dimension one or 
two invariant under stf (unless = 0, in which case we are done). Since ir is irreducible, 
the space W spanned by C C (G) • L is dense in J^. Therefore, P(W) = srf ■ L C L shows 
that L = Jtff K , finishing the proof. □ 

Proof of Theorem \5.3[ Since K° = SO(n) is compact, it must fix a point in the symmetric 
space X(p, oo). This means that there is a positive definite, p-dimensional subspace 
V C Jt? which is A'°-invariant. The smallest dimension of a non-trivial representation 
of K° being n, the hypothesis p < n implies that the action of K° on V is trivial. 
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Since p > 2, this implies that the space Jf K has dimension greater than 2, and the 
representation n cannot be irreducible according to Proposition 15.41 □ 

We conclude with a last proposition. We will say that a representation g : G — > 0(p, oo) 
is geometrically Zariski dense if g(G) does not fix any point at infinity in X(p, oo) and docs 
not preserve any non-trivial closed totally geodesic submanifold. For finite dimensional 
symmetric spaces, this is equivalent to the Zariski density in the usual sense. When p = 1, 
X(l, oo) = H°° and any totally geodesic subspacc is the intersection of H°° with a linear 
subspacc of the corresponding Hilbcrt space (in the hypcrboloid model). In that case a 
representation g is irreducible if and only if it is geometrically Zariski dense. There is no 
such correspondance when H°° is replaced by X(p, oo) (p > 2). However, one still has: 

Proposition 5.5. Let g: G — >• 0(p, oo). 

• If g is geometrically Zariski dense, then g is irreducible. 

• If g is irreducible, g{G) does not fix a point in the boundary of X(p, oo). 

Proof. We first assume that g is geometrically Zariski dense and prove its irreducibility. 
Let V C Ji? be a closed non-zero invariant subspace. We assume by contradiction that 

V Jff. If the restriction of the quadratic form B p to V is non-degenerate, then = 

V © V 1 - (see [1]). Since G has no fixed point in X(p, oo), we can assume, upon replacing 

V by its orthogonal, that B p is not definite on V. Hence it has signature (q, q') for 
some positive integers q < p and q' < +oo. If B p is definite on V , this implies that G 
preserve a totally geodesically embedded copy of the symmetric space X(q,q J ). If B p is 
not definite on V , G preserves a product of two symmetric subspaces X(q, q') x X(r, r'), 
where X(r,r') is associated to V . In any case, this is a contradiction. If on the other 
hand the restriction of B p to V is degenerate, we can assume that V is isotropic upon 
replacing it by VD V- 1 . According to Proposition 6.1 in [TB], this implies that G preserves 
a point in the boundary at infinity of X(p,oo). This is a contradiction again. 

The second part of the proposition follows once again immediately from Proposition 6.1 
in [16]. □ 
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